Non-Contiguous 3D LIDAR Imaging Of Targets With Complex Motion

ABSTRACT

A plurality of scattered laser pulses is received, each scattered laser pulse associated with at least one plurality of respective dwells. A set of 3D velocity information is received, which is derived from photo events detected in information associated with the received plurality of scattered laser pulses. Each dwell in the plurality of dwells is associated with one or more photo events. The 3D velocity information comprises information estimating each respective photo event’s respective position in 6D space during the respective dwell associated with the photo event. For each dwell, its respective photo events are projected into a common reference frame, determined based on the 3D velocity information, to generate a set of motion-compensated point clouds. Each respective motion-compensated point cloud, for each dwell, is registered to the other motion-compensated point clouds in the set, to generate a set of registered point clouds, which are merged into a volumetric image.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

Certain embodiments described herein were made with Government support.The U.S. Government may have certain rights in certain embodimentsdescribed herein.

FIELD

Embodiments of the disclosure generally relate to devices, systems, andmethods for signal processing associated with LADAR systems. Moreparticularly, the disclosure describes embodiments relating to devices,systems, and methods that use LADAR imaging for constructing threedimensional images of targets having complex and unknown motion.

BACKGROUND

LADAR (laser detection and ranging) involves a technology that useslight, typically laser technology, to measure distances (range), speed,at least some atmospheric parameters, can also capture high resolutionimaging information, and can locate and characterize targets. In someinstances, the term “LiDAR” can be more used in applications relating tomapping terrain or collecting information about the atmosphere, whereas“LADAR” can be used more in applications relating to locating andcharacterizing smaller point targets like vehicles or other manufacturedobjects, but this is not limiting. Either way, it is the same basictechnology, only the type of target being ‘ranged’ is different. In thepresent application, any use of the term “LADAR” is intended also toencompass “LiDAR,” as will be understood.

LADAR operates in a manner not unlike radar but beams of laser light areused instead of radio waves. In particular, LADAR generally usesultraviolet, visible, or near infrared light to image objects orterrains. A LADAR system measures distances to objects/terrain byilluminating the objects with light and measuring the reflected pulseswith a sensor. A laser is one example of a light source that can be usedin a LADAR/LiDAR system. Using a narrow laser beam, for example, aLADAR/LiDAR system can detect physical features of objects withextremely high resolutions. A LADAR can generate point clouds of adesired region in the environment. Thus, LADAR has been used to createhigh resolution survey maps of geographic areas and detailedthree-dimensional (3-D) images of objects.

LADAR also can be used to characterize moving targets and range totargets. For example, pulsed LADAR systems provide an active sensingsystem that can determine the range to a target by measuring the time offlight (“ToF”) of short laser pulses reflected off the target. LADARsystems may be used to characterize a moving target, e.g., a targetmoving toward or away from the LADAR system, and also to capturehigh-resolution imaging information about moving and stationary targets.More recently, LADAR has been implemented to support control andnavigation of various types of obstacle avoidance systems, such asautonomous, intelligent vehicle systems (e.g., driverless and/orself-driving vehicles) and active collision avoidance systems invehicles with drivers. The safety and success of autonomous vehiclesdepends at least in part on their ability to accurately map and respondto their surroundings in real time. Further, an accurate perception ofmoving and stationary objects, and also of motion of moving objects, canbe important for successful operation of active collision avoidancesystems as well autonomous vehicle systems.

SUMMARY

The following presents a simplified summary in order to provide a basicunderstanding of one or more aspects of the embodiments describedherein. This summary is not an extensive overview of all of the possibleembodiments and is neither intended to identify key or critical elementsof the embodiments, nor to delineate the scope thereof. Rather, theprimary purpose of the summary is to present some concepts of theembodiments described herein in a simplified form as a prelude to themore detailed description that is presented later.

The processing methods used to generate a high-resolution image usingsome types of LADAR systems may have various shortcomings. For example,multi-dwell imaging of static scenes is a common feature of LADARsystems used for wide-area mapping. These systems register data fromdisparate “looks” (also referred to herein as “dwells”) at the area ofinterest, to correct for navigation and pointing knowledge errors.However, such systems cannot account for in-scene motion between dwells,so in some instances, all non-static content is smeared and/or lost inthe merged product, when a final image is generated. Many radar andother systems use motion compensation techniques to undo the effects ofwithin-dwell motion to increase signal to noise ratio (SNR) for targetdetection and avoid smear in image formation. However, these motioncompensation approaches are valid only for measurements made over thecourse of a single contiguous dwell. In many cases, they are alsolimited to systems capable of coherent sensing.

At least some embodiments herein attempt to address at least some ofthese limitations. For example, at least some embodiments herein combinethe advantages of both methods (multi-dwell imaging and motioncompensation) for the application of imaging dynamic targets withdirect-detect LADAR systems. Certain embodiments herein adapt and add toa single-dwell acquisition algorithm disclosed in commonly assigned U.S.Pat. Application No. 17/138,386, having attorney docket number RAY-376US(20-14113-US-NP), entitled “SIX DIMENSIONAL TRACKING OF SPARSE LADARDATA” (hereinafter “ ‘386 application”) and filed on Dec. 30, 2020,which shares two inventors in common with the present application, whichapplication is hereby incorporated by reference. This application alsoreferences and incorporates by reference a 4D target tracker describedin another copending and commonly assigned U.S. Pat. Application No.17/138,365, having attorney docket number RAY375US (20-13112-US-NP),entitled “VIDEO TRACKING OF SPARSE GEIGER-MODE DATA” and filed on Dec.30, 2020, which shares one inventor in common with the presentapplication, (hereinafter “ ‘365 application”), which application ishereby incorporated by reference.

In certain embodiments herein, motion-compensated target returns (e.g.,motion-compensated point clouds) are extracted from each individualdwell then, these motion-compensated point clouds are corrected forrigid translation and orientation errors via dwell-to-dwell registrationto generate higher-quality multi-look imagery. That is, registering eachrespective motion-compensated point cloud is configured to correct atleast one of translation and orientation errors.

In a first aspect, a system is provided, comprising a receiver and aprocessor. The receiver is configured to receive a plurality ofscattered laser pulses, each respective scattered laser pulse in theplurality of scattered laser pulses associated with at least onerespective dwell. The processor is configured for: receiving a set ofthree-dimensional (3D) velocity information derived from photo eventsdetected in information associated with the received plurality ofscattered laser pulses, wherein each dwell in the plurality of dwells isassociated with one or more respective photo events, and wherein the 3Dvelocity information comprises information estimating each respectivephoto event’s respective position in six-dimensional (6D) space duringthe respective dwell associated with the respective photo event;projecting, for each dwell, the respective photo events of the dwellinto a common reference frame, wherein the common reference frame isdetermined based on the 3D velocity information, to generate a set ofmotion-compensated point clouds, the set of motion-compensated pointclouds comprising at least one motion-compensated point cloud for atleast one respective dwell; registering each respectivemotion-compensated point cloud, for each dwell, to the othermotion-compensated point clouds in the set of motion-compensated pointclouds, to generate a set of registered point clouds; and merging theset of registered point clouds into a volumetric image.

In some embodiments, the processor is further configured for: computing,for the volumetric image, a local spatial point density; and applying anon-linear scaling to the local spatial point density to form a scaledvolumetric image. In some embodiments, the processor is furtherconfigured for displaying the scaled volumetric image. In someembodiments, the plurality of dwells is noncontiguous. In someembodiments, the plurality of dwells is separated in time. In someembodiments, wherein the set of 3D velocity information is generatedusing a state space carving (SSC) process.

In some embodiments, wherein the plurality of scattered pulses isassociated with a single photon laser detection and ranging (LADAR)(SPL) system. In some embodiments, the scattered laser pulses anddetected photo events are associated with a target and wherein thecommon reference frame comprises an instantaneous reference frame thatis associated with the target and which is based on the set of 3Dvelocity information. In some embodiments, the target associated withthe plurality of scattered laser pulses, has smoothly differentiablemotion.

In some embodiments, the processor is configured for registering eachrespective motion-compensated point cloud to correct at least one oftranslation and orientation errors, in at least a portion of the set ofmotion-compensated point clouds. In some embodiments, the processor isconfigured for registering each respective motion-compensated pointcloud using an iterative closest point (ICP) algorithm.

In another aspect, a method is provided. A plurality of scattered laserpulses is received, each respective scattered laser pulse in theplurality of scattered laser pulses associated with at least oneplurality of respective dwells. A set of three-dimensional (3D) velocityinformation is received, the set of 3D velocity information derived fromphoto events detected in information associated with the receivedplurality of scattered laser pulses, wherein each respective dwell inthe plurality of dwells is associated with one or more respective photoevents, and wherein the 3D velocity information comprises informationestimating each respective photo event’s respective position insix-dimensional (6D) space during the respective dwell associated withthe respective photo event. For each dwell, its respective photo eventsare projected into a common reference frame, wherein the commonreference frame is determined based on the 3D velocity information, togenerate a set of motion-compensated point clouds, the set ofmotion-compensated point clouds comprising at least onemotion-compensated point cloud for at least one respective dwell. Eachrespective motion-compensated point cloud, for each dwell, is registeredto the other motion-compensated point clouds in the set ofmotion-compensated point clouds, to generate a set of registered pointclouds. The set of registered point clouds are merged into a volumetricimage.

In some embodiments, the method further comprises displaying thevolumetric image as a scaled volumetric image. In some embodiments, themethod further comprises generating the set of 3D velocity informationusing a state space carving (SSC) process. In some embodiments, thescattered laser pulses and detected photo events are associated with atarget and wherein the common reference frame comprises an instantaneousreference frame that is associated with the target and which is based onthe set of 3D velocity information. In some embodiments, the registeringeach respective motion-compensated point cloud is configured to correctat least one of translation and orientation errors, in at least aportion of the set of motion-compensated point clouds.

In another aspect, a means for laser detection is provided, comprising:means for receiving a plurality of scattered laser pulses, means forreceiving a set of three-dimensional (3D) velocity information derivedfrom photo events detected in information associated with the receivedplurality of scattered laser pulses, means for projecting, for eachdwell, the respective photo events for the dwell into a common referenceframe, means for registering each respective motion-compensated pointcloud, for each dwell, to the other motion-compensated point clouds inthe set of motion-compensated point clouds, to generate a set ofregistered point clouds, and means for merging the set of registeredpoint clouds into a volumetric image.

In the means for receiving the plurality of scattered laser pulses, eachrespective scattered laser pulse in the plurality of scattered laserpulses is associated with at least one plurality of respective dwells.In the means for receiving a set of 3D velocity information, each dwellin the plurality of dwells is associated with one or more respectivephoto events, and wherein the 3D velocity information comprisesinformation estimating each respective photo event’s respective positionin six-dimensional (6D) space during the respective dwell associatedwith the respective photo event. In the means for projecting, for eachdwell, the respective photo events of the dwell into a common referenceframe, the common reference frame is determined based on the 3D velocityinformation, to generate a set of motion-compensated point clouds, theset of motion-compensated point clouds comprising at least onemotion-compensated point cloud for at least one respective dwell.

In some embodiments, the means for laser detection of claim furthercomprises means for generating the set of 3D velocity information usinga state space carving (SSC) process. In some embodiments, the scatteredlaser pulses and detected photo events are associated with a target andthe common reference frame comprises an instantaneous reference framethat is associated with the target and which is based on the set of 3Dvelocity information. In some embodiments, the registering eachrespective motion-compensated point cloud is configured to correct atleast one of translation and orientation errors, in at least a portionof the set of motion-compensated point clouds.

It should be appreciated that individual elements of differentembodiments described herein may be combined to form other embodimentsnot specifically set forth above. Various elements, which are describedin the context of a single embodiment, may also be provided separatelyor in any suitable sub-combination. It should also be appreciated thatother embodiments not specifically described herein are also within thescope of the claims included herein.

Details relating to these and other embodiments are described more fullyherein.

BRIEF DESCRIPTION OF THE DRAWINGS

The advantages and aspects of the described embodiments, as well as theembodiments themselves, will be more fully understood in conjunctionwith the following detailed description and accompanying drawings, inwhich:

FIG. 1 is a block diagram of a LADAR system, in accordance with oneembodiment;

FIG. 2 is an exemplary model of a target with complex motion, inaccordance with one embodiment;

FIG. 3 is an exemplary model of a trajectory of the target of FIG. 2 ,which trajectory is unknown a priori, in accordance with one embodiment;

FIG. 4A is a flowchart 400 showing further details of an exemplary statespace carving (SSC) process usable as part of one of the processingblocks in the flowchart of FIG. 6 , in accordance with one embodiment;

FIG. 4B is a first table showing raw photo events and pulses associatedwith two exemplary targets, as generated by the SSC method of FIG. 4A,in accordance with one embodiment

FIG. 4C is a second table showing the 6D state space positions of thetwo targets of FIG. 4B, as generated using the SSC method of FIG. 4A, inaccordance with one embodiment;

FIG. 4D is a first exemplary image of a single dwell, based at least inpart on the method of FIG. 4A and the target information in FIGS. 4B-4Cillustrating an exemplary target trajectory and outputs in a singledwell;

FIG. 4E is an image depicting object facet models generated based on theimage of FIG. 4D, for a single dwell, in accordance with one embodiment;

FIG. 5A is an exemplary block diagram illustrating overall processingflow of a system that implements multi-dwell processing, in accordancewith one embodiment;

FIG. 5B is an exemplary block diagram illustrating at a high levelcertain sub-processes in the multi-dwell processing block of FIG. 5A;

FIG. 6 is a flowchart illustrating a method that operates in the systemof FIG. 1 and in accordance with the block diagrams of FIGS. 4A and5A-5B;

FIG. 7 is a set of graphs illustrating the operation of the method ofFIG. 6 to produce single-dwell state estimates, in accordance with oneembodiment;

FIG. 8A is a first exemplary merged point cloud image of a single dwellimage, in accordance with one embodiment

FIG. 8B is a second merged exemplary point could image of a multi-dwellimage, in accordance with one embodiment;

FIG. 9A is a first exemplary point cloud image of a single dwell imagewith object facet model;

FIG. 9B is a second exemplary point cloud image of a multi-dwell imagewith object facet model, in accordance with one embodiment; and

FIG. 10 is a block diagram of an exemplary computer system usable withat least some of the systems and apparatuses of FIG. 1-9B, in accordancewith one embodiment.

The drawings are not to scale, emphasis instead being on illustratingthe principles and features of the disclosed embodiments. In addition,in the drawings, like reference numbers indicate like elements.

DETAILED DESCRIPTION

The following detailed description is provided, in at least someexamples, using the specific context of LADAR systems target detectionsystems configured to detect, track, monitor, and/or identify terrainand/or targets, where targets can include (but are not limited to)aircraft (both unmanned and manned), unmanned aerial vehicles, unmannedautonomous vehicles, robots, ships, spacecraft, automotive vehicles, andastronomical bodies, or even birds, insects, and rain. At least someembodiments herein are usable with any systems involved with any radarapplications, including but not limited to military radars, air trafficcontrol radars, weather monitoring radars, etc.

Unless specifically stated otherwise, those of skill in the art willappreciate that, throughout the present detailed description,discussions utilizing terms such as “opening”, “configuring,”“receiving,”, “detecting,” “retrieving,” “converting”, “providing,”,“storing,” “checking”, “uploading”, “sending,”, “determining”,“reading”, “loading”, “overriding”, “writing”, “creating”, “including”,“generating”, “associating”, and “arranging”, and the like, refer to theactions and processes of a computer system or similar electroniccomputing device. The computer system or similar electronic computingdevice manipulates and transforms data represented as physical(electronic) quantities within the computer system’s registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission, or display devices. The disclosedembodiments are also well suited to the use of other computer systemssuch as, for example, optical and mechanical computers. Additionally, itshould be understood that in the embodiments disclosed herein, one ormore of the steps can be performed manually.

Before describing in detail the particular improved systems, devices,and methods, it should be observed that the concepts disclosed hereininclude but are not limited to a novel structural combination ofcomponents and circuits, and not necessarily to the particular detailedconfigurations thereof. Accordingly, the structure, methods, functions,control and arrangement of components and circuits have, for the mostpart, been illustrated in the drawings by readily understandable andsimplified block representations and schematic diagrams, in order not toobscure the disclosure with structural details which will be readilyapparent to those skilled in the art having the benefit of thedescription herein.

The following description includes several terms for which thedefinitions are generally known in the art. However, the followingglossary definitions are provided to clarify the subsequent descriptionand may be helpful in understanding the specification and claims.

“Point Cloud” at least includes a set of data points in 3-D space, whichtogether represent a 3-D shape or object. Each point in the data set ofa point cloud, in certain embodiments, is represented by an x, y, and zgeometric coordinate. Point clouds provide a way to assemble a largenumber of single spatial measurements into a dataset that can berepresented as a describable object. Point cloud processing is used invarious applications, including LADAR/LIDAR, robot navigation andperception, depth estimation, stereo vision, visual registration, inadvanced driver assistance systems (ADAS) and autonomous navigationsystems.

“LADAR system” (also known in the art as LIDAR system; the terms LADARand LiDAR are used interchangeably herein) broadly includes at least anysystem that can determine values of parameters indicative of a distancebetween a pair of tangible objects, or the depth of a region within atangible object, whether or not either of the objects in the pair ismoving, based on reflected light. The tangible objects can be any typeof entity or thing that light can reflect off of, whether fixed ormovable, solid or liquid, including but not limited to humans, animals,reptiles, birds, vehicles (including those traveling on land, in theair, in space, and in or on water), water and other liquids, both insolid and liquid form, buildings, structures, plants, inanimate objects(whether natural or man-made), objects under automated and/or remotecontrol, and/or objects under control of a person. In at least someembodiments, at least some of the LADAR systems and methods describedherein are configured to determine or construct a high signal to noiseratio (SNR) three-dimensional (3D) image of a target, where the targethas unknown and possibly complex motion. At least some embodimentsherein relate to LADAR systems and method capable of determining adistance between a pair of tangible objects, determine a direction oftravel of one or more tangible objects, and/or determine a velocity ofone or tangible objects, based on reflections of light emitted by theLADAR system, where the direction, range, and/or velocity determinationscan be absolute or relative and can broadly include generating outputswhich are indicative of at least one or more of distances between pairsof tangible objects, velocity of one or more tangible objects, and/oracceleration of one more tangible objects (including negativeacceleration).

“Light source” at least refers to any device configured to emit light,including but not limited to lasers such as gas lasers, laser diodes,solid-state lasers, high power lasers, and the light can be emitted inmany different spatial and temporal formats, including but not limitedto pulses (including short and long pulses having detectable rise andfall times), sequences of pulses in the form of one or more dwells,bursts, organized point clouds, random spatial patterns, etc. In someembodiments, the emitted light is at a wavelength between about 650 nmand 1150 nm. Alternatively, the light source may include a laser diodeconfigured to emit light at a wavelength between about 850 nm and about1050 nm (NIR), or between about 1300 nm and about 1600 nm (SWIR).

“Time of Flight,” (ToF) at least refers at least to a period of timebetween the emission of a light signal (also referred to herein as alight beam or photons) from a light source, the light beam striking thetangible object, and its return for detection by the sensor in areceiver system. In some embodiments, the sensors of the LADAR systemconvert the received signal into temporal information indicative of ToF.For example, by using known information (such as the speed of light inthe medium of travel, e.g., air), the ToF information is processed togenerate information about a distance the light signal traveled betweenemission and detection.

“Single Photon LADAR” (SPL) (also known in the art as “single photoncounting LADAR,” “photon counting detector,” “single photon detector,”“single photon avalanche diode (SPAD)-based pulsed LADAR,” ) refers atleast to types of LADAR that are capable of measuring/detecting lightdown to the smallest physically possible increment: a photon. Note thatSPL systems are not necessarily only capable of detecting a singlephoton, but they can at least be capable detecting a single photon.“Geiger-mode LiDAR,” (also referred to in the art as “Geiger modedetectors” are a subset of single photon detectors and are explicitlyonly capable of detecting a single photon at a time. As will beappreciated, there are various types of single-photon detectiontechnologies. SPL, in accordance with at least some embodiments herein,at least refers to a form of LADAR that uses detectors that require onlyone detected photon per ranging measurement, as opposed to hundreds orthousands of detected photons per ranging measurement for conventionallinear mode airborne LADARs. That is, SPL is capable ofdetecting/measuring the ToF of individual photons. Single Photon Lidar(SPL) provides a high-density point cloud that can be acquired from ahigh altitudes. Known linear-mode-LADAR (LML) systems can recordmultiple returns (commonly 5 returns per pulse) of energy for each laserpulse (although the number of photons required to trigger a return canbe proprietary to the instrument being used to process returns); in someLML system, hundreds or thousands of photons may be required to triggera return in order to reduce the impact of noise. In sharp contrast, inan SPL system, a single photon can trigger a return at the sensor for anSPL system. Thus, SPL can provide an efficient approach to rapid,high-resolution 3D mapping in terms of swath, spatial resolution,acquisition time and density of range returns. SPL has emerged as anoptimal for depth imaging through challenging environments, especiallywhen high point densities are required over very large areas, or whenimprovements in measurement rates can significantly reduce dataacquisition costs. For example, SPL has been used in such challengingscenarios as imaging though highly scattering underwater conditions,free-space imaging through obscurants such as smoke, forest cover,and/or fog, and depth imaging of complex multiple surface scenes. WithSPL, the point density on the ground can be 10-100 times higher for SPLdata than that obtained with multi-photon systems at the same flightaltitude, which can help to reduce operation costs. In addition, thelong-range capabilities (kilometers), excellent depth resolution(centimeters), and use of low-power (eye-safe) laser sources renders SPLa strong candidate for use in applications such as autonomous vehicles.Certain embodiments herein advantageously use SPL. Commonly assignedU.S. Pat. 9,335,414 (“FREQUENCY AGILE LADAR”), which is herebyincorporated by reference, describes an illustrative single photoncounting LADAR system that can be adapted to be usable with at leastsome embodiments herein.

Turning now to the drawings, in which like reference characters indicatecorresponding elements throughout the several views, attention is firstdirected to FIG. 1 , which is a block diagram 100 of an exemplary LADARsystem 102, in accordance with one embodiment, which is configured toaddress at least some of the aforementioned shortcomings.. The exemplaryLADAR system 102 is usable to construct a three dimensional (3D) imageof a target 104 ( which is shown for illustrative purposes only to be anaircraft but this is not limiting) , wherein in some embodiments the 3Dimage is generated based at least in part on 3D velocity informationderived from photo events (e.g., one or more photo events) detected in aplurality of scattered laser pulses received at the receiver 108, or ininformation associated with one or more photo events, and is usable toprovide that 3D image to be displayed as a scaled volumetric image onany type of an output device 112, where the output device 112 can beanother processing system, another entity (which can be a program,machine, and/or human), a display or other device viewable by a user,etc. In some embodiments, the output device 112 can be part of exemplaryLADAR system 102 itself. The 3D velocity information, in someembodiments, includes information estimating each photo event’s positionin six-dimensional (6D) space during its respective dwell. The exemplaryLIDAR system 102 includes four subsystem: a laser transmitter 106, areceiver 108 (e.g., an optical receiver), a processor 110, and asix-dimensional (6D) tracking system 114 (which can be implementedentirely by the processor 110, as will be understood.

The laser transmitter 106 is configured to generate generating laserpulses 122 (e.g., pulses of photons/light pulses 122) when commanded(e.g., by processor 110), and to direct these pulses 122 in a desireddirection (e.g., the direction of the target 104). In certainembodiments, one characteristic of these pulses is that they are each1-25 nanoseconds in duration, so that the resulting range measurementmay be accurate to within a few meters. In some embodiments, the lasertransmitter 106 is configured to transmit pulses 122 at a desired pulserepetition frequency (e.g., 20 kHz).

The receiver 108, in certain embodiments, includes an optical system(not shown, but well understood in the art) capable of collectingreflected light 124 (which includes light returns that arrive inresponse to transmitted laser pulses ) from the target 104, and aphotodetector (not shown, but well understood in the art) within thereceiver 108, which is capable of recording the arrival time of incominglight (e.g., reflected light 124), as will be understood in the art. Thephotodetector is capable of timing the arrival of return pulses with anaccuracy similar in scale to the laser pulse duration. In someembodiments, the receiver 108 includes elements such as a focal planearray (not shown) which may be arranged to receive scattered laserpulses, as will be understood in the art. The receiver 108, incooperation with the processor 110, converts the reflected light 124into data 126 that it provides to the 6D tracking system 114.

The processor 110 can be any type of computing device capable ofcontrolling the operation of the laser transmitter 106 and receiver 108,and to concurrently (e.g., simultaneously) extract video/photoinformation about the target 104 from the detections made by thereceiver 108, such as range and range rate of the target 104, based onthe transmit and return times (“time of flight”) of the photons 122 andreflected light 124. In certain embodiments, the processor 110 isconfigured to implements the 6D tracking system 114, described furtherherein, to perform processing on the data 126 (which can include videodata, such as video input data ) from receiver 108, such as extractedvideo/photo information. In certain embodiments, a computer system, suchas that shown and described further herein in connection with FIG. 10 ,is usable to implement the processor 110.

The processor 110, in certain embodiments, also is configured tosynchronize commands issued to the laser transmitter 106 and thereceiver 108. In certain embodiments, an essential aspect of the returnprocessing of the returns of reflected light 124 received at receiver108 is the ability to determine target characteristics accuratelydespite complex scattering of the transmitted light pulses 122 imperfectdetection of the reflected light 124 , unwanted detections due toambient light and electrical noise, modulation of the return due tomotion of the target 104, and complex and unknown motion of the target104, among other practical complications. In certain embodiments, the 6Dtracking system 114 assists in the determination of one or morecharacteristics of the target 104. In certain embodiments, a multi-dwellprocess (described further below) helps to produce a high signal tonoise ratio (SNR) 3D image of the target 104, in the presence ofcomplect and unknown motion of the target 104 . This is describedfurther herein.

The 6D tracking system 114, in some embodiments, includes a state spacecarving (SSC) subsystem 117 and a multi dwell subsystem 120, each ofwhich are explained further herein. The SSC subsystem, in certainembodiments, incorporates an SSC method which is more particularlyexplained in the aforementioned ‘386 patent. The SSC subsystem 117,described further herein, include a 2D target tracking detectorsubsystem 116 (which includes a 2D target tracking detector) and 4Dtarget tracking detector subsystem 118. The SSC subsystem 117 shown inFIG. 1 is but one illustration of a type of system that can produce therequired 3D velocity outputs 136, and it is envisioned that othersystems capable of producing the same types of outputs based on thereceived data 126, are usable, as explained further herein. For example,an acquisition and motion estimation algorithm subsystem could beadapted by those of skill in the art to produce the same outputsdescribed herein for the SSC subsystem 117, such as tagged photo eventsthat are assigned a position in 3D space.

To help understand applicability of the embodiments described herein, itis useful to first consider the situation of a complex target modelhaving a dynamic 3D trajectory, with complicated motion. FIG. 2 is anexemplary model 100 of a complex target 202 with complex motion, inaccordance with one embodiment, and FIG. 3 is an exemplary model of atrajectory 300 of the complex target 202 of FIG. 2 , which trajectory300 is unknown a priori, in accordance with one embodiment. The complextarget 202 of FIG. 2 is an occluding bi-sphere complex (center figure,labeled as complex target 202), with an exemplary front-and-backcomponent laser radar scattering cross section (LRCS) ratio of 3-to-1.As shown in FIG. 3 , the trajectory 300 of the complex target 202 (whichis not known in advance) is a spiraling and accelerating helix, as shownin FIG. 3 . The total signal-to-noise ratio is extremely low, with ~350total signal events and ~850,000 background events per 150 ms dwell.Note that with this SNR, the complex target 202 would be undetectablewith conventional 2D processing (e.g., shift and sum (SAS) maps).

The aforementioned ‘386 and ‘365 applications provided further details(which are also summarized herein) on several techniques that are ableto detect lower-SNR targets than was previously possible. In certainembodiments herein, additional techniques are provided to providefurther ways to detect low SNR targets, especially those having complexand unknown motion, and some of these embodiments leverage, apply, andadapt the techniques of the aforementioned ‘386 and ‘365 applications,as part of the embodiments. In certain embodiments herein, thetechniques of the aforementioned ‘386 and ‘365 patent applications arenot necessary, and alternate techniques are employed.

In certain embodiments herein, a technique of multi-dwell processing isprovided to enable non-contiguous 3D LADAR imaging of targets withcomplex motion. Some of these embodiments use the aforementioned SSCmethod of the ‘386 patent, , to help provide 3D velocity information forthe multi dwell processing. SSC, in certain embodiments, achieves betterdetection statistics by iteratively projecting raw LADAR data across twoorthogonal sub-spaces of the full 6D state-space - first into 2D (R, Ṙ)(i.e., range, range-rate), then 4D (az (azimuth), el (elevation), az-dot(azimuth rate), el-dot (elevation rate), also referred to by therespective characters ϕ, θ, ϕ, θ )).

To further the understanding of the embodiment herein, FIG. 4 is aflowchart showing further details of an exemplary state space carving(SSC) process 400 usable as part of one of the processing blocks in theflowchart of FIG. 6 (discussed further herein), in accordance oneembodiment. The SSC process 400 of FIG. 4 , as described further below,is able to detect lower-SNR targets than was previously possible..Within each subspace, statistically significant clusters of events aretagged and retained, while all other events are discarded. Stochasticvariation can cause background events to randomly cluster in eitherspace, thereby masquerading as signal - but due to the orthogonality ofthe two subspaces, it is unlikely for such a false cluster to beretained through a full SSC iteration. As a result, the SSC process ofFIG. 4A “carves” away background as it iterates. Rigid-body targets, onthe other hand, are compact across all six dimensions of state, andtherefore signal photo-events are retained throughout the carvingprocess

Referring to FIGS. 1 and 4A, at block 405, the receiver 108 providesvideo input data 126 to the 6D tracking system 114, where the videoinput data includes target photo events and background photo events,where the video data 126 may be derived from a LADAR dwell of a targetand the target surroundings. Video data 126 may be derived from receivedor scattered laser pulses, the laser pulses transmitted by the lasertransmitter 106 or the like. In certain embodiments, the video data 126may include sparse and binary video data such as Geiger-Mode AvalanchePhotodiode data (GMAPD). In certain embodiments, the video data is as aresult of a single photon detector that is capable of detectingindividual photons in a laser transmission.

In certain implementations, the 2D target tracking detector 116 maythereafter receive video data 126, so that the 2D target trackingdetector 116 can transform the photo events into a 2D target trackingarray including range and range rate(R, Ṙ) parameters (block 410). The2D target tracking detector 116 may be operated by use of processor 110or by an external exemplary computer system (not shown), similar to thatof FIG. 10 . The 2D target tracking detector 116 may be configured toconcurrently determine the range and the range-rate (i.e., velocity) ofphoto events within video data 126 based on transmissions of laserpulses 122 and received reflected light 124 such as return times ofphotons. Henceforth, the terms speed, velocity, and range-rate refer tothe velocity of the target 104 relative to the exemplary LADAR system102 along the range axis (i.e., the line/direction connecting theexemplary LADAR system 101 and the target 104). The 2D target trackingdetector 116 may accurately determine these target 104 characteristicsdespite complex scattering of the transmitted light, imperfect detectionof the returns, unwanted detections due to ambient light and electricalnoise, modulation of the return due to target motion, and/or otherpractical complications and limitations.

In some implementations, the 2D target tracking detector 116 scales(e.g., stretches or compresses) the transmit times of emitted laserpulses 122 according to a plurality of hypothesized and/or predictedvelocities and, for each hypothesized velocity, computes across-correlation of the scaled transit times with the return times ofdetection events, and identifies the peak cross-correlation power valuefor the plurality of hypothesized/trial velocities. Determining thetemporal scaling that yields the highest correlation peak value allowsthe 2D target tracking detector 116 to concurrently (e.g.,simultaneously) determine both the range and range-rate of photo events.An example 2D target tracking detector that determines both the rangeand range-rate of photo events is described in commonly assigned U.S.Pat. Application No. 16/863064 (inventors Greenberg & Marcus, which isthe same as the present application) entitled “SYSTEM AND METHOD FORDETERMINING RANGE-RATE AND RANGE EXTENT OF A TARGET,” filed on Apr. 30,2020 and published on ______ (hereinafter “‘064 application”). Thecontent of the ‘064 application, particularly the content related to theprocess of target acquisition (e.g., FIG. 3 of U.S. Pat. Application No.16/863,064 and the accompanying description), is incorporated herein byreference. However, it will be understood by those of skill in the artthat other 2D target tracking detectors are, of course, usable.

Referring still to FIGS. 1 and 4A, in some implementations, the 2Dtarget tracking detector 116 may calculate the range and the range-rateof the target in video data 126 based on a plurality ofcross-correlation power values, wherein the 2D target tracking detector116 identifies a peak cross-correlation power value (e.g., the highestcross-correlation value) from among the plurality of cross-correlationpower values and determines the pair-wise difference value associatedwith the peak cross-correlation power value.

In block 415 of FIG. 4A, photo events that are determined to bebackground events (i.e., not associated with detections) are discarded,and photo events determined to be 2D target events are tagged and areexported (block 420) to the 4D target tracking detector 118. Inparticular, after determining the range and range-rate of the target 104within video data 126 or attempting to find at least one peakcorrelating to a statistically significant result within atwo-dimensional array with range and range-rate dimensions, video data126 (with tags) may thereafter be transmitted to the 4D target trackingdetector 118 (block 420). In certain embodiments, statisticallysignificant results that are sent to the 4D target tracking detector 118may reach a prescribed threshold of counts. In some embodiments, photoevents associated with every range and range rate detection within videodata 126 may be applied to the 4D target tracking detector 118. In someembodiments, video data 126 not associated with detections may bediscarded or otherwise rejected before the adjusted and tagged videodata 128 is exported to 4-D target tracking detector 118

In some embodiments, the 4D target tracking detector 118 may cross-rangefilter the photo events, accounting for both focal plane position andfocal plane motion over a LADAR dwell. In block 425, the tagged photoevents (received from the 2D target tracking detector 116) aretransformed into a 4D target array, including azimuth, elevation,azimuthal velocity, and elevation velocity (ϕ, θ, ϕ, θ). Various typesof 4D video trackers are usable as the 4D target tracking detector 118,as will be appreciated. In some implementations, 4D target trackingdetector 118 may be operated by use of processor 110 or via an externalcomputer system (not shown), as in FIG. 10 . In some embodiments, the 4Dvideo tracker can be implemented by configuring the processor 110 orexternal computer system, to generate a video filter array, the videofilter array (VFA) including a set of estimated velocity pixelcoordinate components arranged in a linear data set while representing aplurality of two-dimensional arrays associated with a plurality offrames, wherein, in these embodiments, the VFA may be stored in a memoryand/or mass storage (not shown), as will be understood by those of skillin the art. In certain embodiments, each of the plurality oftwo-dimensional arrays may have dimensions equal to dimensions of thefocal plane array of the receiver 108 and may generate a plurality ofdetected photo events based on received scattered laser pulses or videodata.

The 4D target tracking detector 118, in certain embodiments, may alsofilter the plurality of photo events transmitted to it by linearlyindexing each of the plurality of detected photo events based on, foreach detected photo event, a vertical position in the focal plane array,a horizontal position in the focal plane array, a frame number, and thedimensions of the focal-plane array (block 430). For example, in someembodiments, the 4D target tracking detector 118 may discard photoevents determined to be background photo events, and tag photo eventsdetermined to be 4D target events (block 430), but this is not required.The 4D target tracking detector 118, in some embodiments, may map eachdetected photo event to a set of estimated velocity pixel coordinatecomponents based on a time between receiving the scattered laser pulsesand the focal-plane array vertical and horizontal positions of each ofthe detected photo events. In some embodiments, the 4D target trackingdetector 118 may store in memory parameters associated with each tagged4D photo event (block 435).

The processor 110 may cause the 6D tracking system 114 to repeat blocks410-435 for a plurality of iterations, until a predetermined or requirednumber of iterations are complete (block 440). In one embodiment, afterthe first iteration, the photo events transformed at block 410 (2Dtarget tracking detector) are the tagged 4D target signal photo eventsfrom block 425, as shown in block 445 of FIG. 4 . The parametersassociated with each of the tagged 4D target signal photo events may bestored in a memory (block 435) . The parameters stored in a memory mayrepresent a six dimensional (6D) array. The parameters may includerange, range-rate, azimuth, azimuthal velocity, elevation, and elevationvelocity. In certain embodiments, as further described herein, 3Dvelocity parameters are provided to a multi-dwell process, describedfurther in FIG. 5A-9 herein.

In some embodiments, the 4D target tracking detector 118 may generate amotion-compensated image associated with the mapped plurality ofdetected photo events in a filtered two-dimensional array havingdimensions equal to the dimensions of the focal plane array. Furtherdetails regarding an implementation of an exemplary 4D tracker aredescribed in co-pending U.S. Pat. Application No. 17/138,365, entitled“VIDEO-TRACKING OF SPARSE GEIGER MODE DATA”, filed on Dec. 30, 2020(hereinafter “the ‘365 patent”), particularly the content related togenerating a video filter array and using a video filter array withGeiger-mode video data (e.g., FIG. 5A-9 and the accompanyingdescription), which application is hereby incorporated by reference.

As a result of the process of FIG. 4A, an in particular the 4D targettracking detection blocks 420-450, four-dimensional filtered and motioncompensated focal plane array images may be generated. The 6D trackingsystem 114 may thereafter associate each four dimensional detection withthe video data that comprises it, and all other video data may bedisposed of or otherwise rejected. With the four dimensional detectiondata output, 6D tracking system 114 may iterate (blocks 440, 445, etc.)wherein the four dimensional detection data output out of 4D targettracking detector 118 is applied to 2-D target tracking detector 116,via feedback loop 132, and subsequently 4-D target tracking detector118. In some embodiments, 6D tracking system 114 may iterate 308multiple times, and in some embodiments, 6D tracking system 114 mayrefrain from iterating 308. The resulting video data 130 may thereafterbe transmitted, exported, or the like to another system for furtherprocessing. For example, in accordance with the embodiments describedfurther herein in connection with FIG. 5A-9, the video data 130 is, incertain embodiments, further processed using a multi dwell subsystem 120(FIG. 1 ) which provides the ability to combine the output frommultiple, noncontiguous dwells of a target with arbitrarily complexmotion, to generate a multi-dwell image. This is discussed furtherherein in connection with FIG. 5A-9 .

Before proceeding to the multi-dwell subsystem and additionalprocessing, an example using and further explaining the SSC process ofFIG. 4A is first discussed. Consider an example of a target thatcompletes its full trajectory after 1 second. The SSC process 400 (alsoreferred to herein as SSC method 400) of FIG. 4A is used to process asingle 150 ms dwell at the beginning of the target trajectory. In someembodiments, after the SSC subsystem 117 processing, the SSC subsystem117 produces three primary outputs (along with several ancillarymetrics, including measurement uncertainties). These outputs include:

-   1. A list of statistically significant targets that were detected    within the data, and their associated 6D state-space position-   2. Pointers to the raw photo-events associated with each target-   3. Pointers to the pulses that correspond to each photo-event

FIG. 4B is a first table 460 showing raw photo events and pulsesassociated with two exemplary targets, as generated by the SSC method ofFIG. 4A, in accordance with one embodiment. In the table 460 of FIG. 4B,the first column is the identification number (ID #) of the target, thesecond column is a set of pointers to raw photo events (s) (Pes) (i.e.,#2 above), and the third column is the set of pointers to the pulsesthat correspond to each photo event (i.e., #3 above).

FIG. 4C is a second table 470 showing the 6D state space positions ofthe two targets of FIG. 4B, as generated using the SSC method of FIG.4A, in accordance with one embodiment. As the table 470 shows, for eachof the two targets, the 1^(st) column is Range position, the 2^(nd)column is X position, the 3^(rd) column is Y position, the 4^(th) columnis Range velocity, the 5^(th) column is X velocity, and the 6^(th)column is Y velocity.

By combining the output of the SSC process 400 of FIG. 4As output withthe raw measured data (e.g., table 460 of FIG. 4B and table 470 of FIG.4C), for the exemplary first and second targets of FIGS. 4B and 4C, itis possible to generate an image from the photo-events that were taggedas signal. Furthermore, because the SSC process 400 includes adetermination of the target’s 3D velocity, it is possible to performthis image-formation within the target’s own reference frame, therebyobtaining a motion-compensated point-cloud, as shown in FIG. 4D, whichis a first exemplary image 480 of a single dwell, based at least in parton the method of FIG. 4A and the target information in FIGS. 4B-4Cillustrating an exemplary target trajectory and outputs in a singledwell. FIG. 4E is an image 490 depicting object facet models of targets1 and 2 of FIG. 4C, generated based on the image of FIG. 4D, and thedata of FIG. 4C, for a single dwell, in accordance with one embodiment.The resulting image (object facet models of FIG. 4E) are roughlyrecognizable as corresponding to the original input bi-sphere model ofFIG. 2 .

The images depicted in the examples FIGS. 4B-4E help to illustrate, inaccordance with some embodiments, the remarkable sensitivity of the SSCmethod of FIG. 4A. This observing scenario – with ~350 signalphoto-events scattered among >240,000x more background events -wouldhave yielded a non-detection with conventional processing techniques.The SSC method of FIG. 4A, however, is able to detect, segment,motion-compensate, and image-form the bi-sphere complex of FIG. 2 .While the SSC method of FIG. 4A provides certain advantages andfeatures, such as background rejection, further testing has shown thatthe output of this algorithm can be used for robust automatic imageformation, in the presence of unknown target motion and high background,across multiple noncontiguous dwells. In certain embodiments herein,systems, methods, and apparatuses are provided that are configured toleverage the novel approach to “detection” that SSC uses and to applythis technique to image formation. Specifically, in certain embodiments,by performing image formation and state-space acquisition simultaneously(or at least substantially simultaneously), the embodiments describedherein allow an observer to collect low-SNR target data, and detect,segment, and image-form simultaneously, even in scenarios where thetarget data were collected across non-contiguous dwells, and/or wherethe target has unknown complex motion.

As will be appreciated, there can be various types of scenarios whichmay force a LADAR system to take data of a single target in anon-contiguous fashion. On example is where a LADAR system mustprioritize a variety of tasks, of which collecting data of the target inquestion is but one. The need to balance these tasks imposes a disjointrevisit schedule onto the target. Another example is where a giventarget has (a priori) unknown, non-linear motion; in such an example,the need to detect the target imposes a maximum contiguous integrationtime, as higher order moments of the unknown motion will smear out thetarget signal over long dwells. In both these examples, the multi-dwellprocess described below, in accordance with some embodiments, allows anobserver to take data from disparate, noncontiguous sub-dwells, andcombine those data to improve the image SNR beyond that of anyindividual sub-dwell. In some embodiments, the combined image qualitywill approach that of a single dwell of a stationary target withintegration time equal to the sum of the integration times over all thesub-dwells. In some embodiments, the final image SNR will differ fromthat of an equivalent contiguous dwell only by factors limiting thedetermination of the target’s instantaneous velocity (i.e., thestate-space resolution). In some embodiments, because of the sensitivityof the SSC process, the multi dwell process can be carried out inscenarios where even detecting the target would be challenging for anystandard, non-state-space based acquisition process.

FIG. 5A is an exemplary block diagram 500 illustrating overallprocessing flow of a system that implements multi-dwell processing, inaccordance with one embodiment. FIG. 5A illustrates a processing flowwhere a plurality of dwells 402A-402N are each processed by a processthat is able to provide 3D velocity as an output, such as respective SSCsub process 404A-404N. The outputs of each of the respective SSC subprocesses 404A-404N are combined into multi-dwell processing sub process406, to generate a multi-dwell image 408. FIG. 5B is a block diagram 550illustrating at a high level certain sub-processes in the multi-dwellprocessing block of FIG. 5A. In certain embodiments, FIG. 5B provides asystem for generating a single motion-compensated 3D image from anarbitrary number of individual LADAR dwells with arbitrary temporalseparations. In certain embodiments, FIG. 5B provides a multidwellprocessing algorithm that combines target acquisition results frommultiple dwells into one merged 3D image. FIG. 6 is a flowchart 600illustrating a method that operates in the system of FIG. 1 and inaccordance with the block diagrams of FIGS. 5A-5B.

Referring to FIGS. 1 and 5A-6, data 402 are received (block 505), wherethe data is based on the 106 transmitter transmitting pulses 122 to oneor more regions, potentially from disparate, noncontiguous sub dwells(e.g., multiple small dwells). Next a process is performed on thereceived data (block 610) to (1) tag photo events within 6-dimensionalspace, and (2) assign each respective tagged photo event/object aposition in 6D space. In certain embodiments, the State Space Carving(SSC) process of FIG. 4A (e.g., SSC 404 of FIG. 5 ) is configured toachieve (1) and (2), as shown in FIG. 6 at block 615, but this is notlimiting. For example, in certain embodiments, the SSC sub-processes404A-404N are configured to use an SSC process (e.g., the SSC process ofFIG. 4A) to estimate 6D state for each dwell, such as via informationestimating each respective photo event’s respective position insix-dimensional (6D) space during its respective dwell. However, thoseof skill in the art will appreciate that other processes may be usableto tag photo events within a 6D space and to assign each respectivetagged photo event a position within 6D space. That is, although atleast some advantageous embodiments use the SSC process of FIG. 4A, itis not required, and any process capable of completing the actions ofblock 610, is usable in accordance with at least some embodimentsherein. For example, in at least some embodiments, instead of the SSCprocess, the actions of block 610 can be completed using any one or moreof a 3D velocity filtering process, SAS maps, coincidence processing,and peak tracking.

As an output of block 610, for each tagged photo event in the set ofphoto events generated in block 610, a 3D velocity determination isreceived (block 620), the 3D velocity determination including, forexample, parameter such as Ṙ, ϕ, θ from the process output of block 610.The dwells are projected into a common reference frame by compensatingfor motion (block 410). For example, each dwell’s photo events areprojected into the target’s own reference frame (instantaneous referenceframe) (block 625), i.e., a common reference frame, , as determined bythe 3D velocity determination of block 620. This results in a set ofmotion-compensated point clouds for each dwell. In block 630, theresulting motion-compensated point clouds for each dwell areauto-registered against one another using a variation of the IterativeClosest Point (ICP) algorithm, wherein the registration (which incertain embodiments is an auto-registration) helps to register pointclouds to refine translation and rotation. This process helps to makeslight adjustments to the point cloud to help minimize the distancebetween the photo events and their corresponding 3D position, basedacross the different dwells, including taking into account rotation ofthe target or receiving source between dwells, as well as arbitrarymotion of the target between dwells. For example, an object may havenatively rotated, or the receiver 108 (or other system) that isreceiving the information, could have changed, such that its “view” ofthe object/target being imaged, may have changed, such that thetarget/object might “appear” to have been rotated. In another example,detected images might be subject to geometric distortion introduced byperspective irregularities wherein the position of the camera(s) withrespect to the scene alters the apparent dimensions of the scenegeometry.

As is known in the art, applying an affine transformation to a uniformlydistorted image can correct for a range of perspective distortions, bytransforming the measurements from the ideal coordinates to thoseactually used. Effectively, in certain embodiments, the autoregistration operation of block 630 performs affine transformation,which, as those of skill in the art will appreciate, helps to correctfor geometric distortions or deformations that occur with non-idealcamera angles.

As those of skill in the art will appreciate, in certain embodiments,commercial mathematical modeling tools, such as Computer Vision Toolbox™(hereinafter “Toolbox,” and available from Math Works of Natick MA), areusable to provide various computerized mathematical tools to helpimplement steps such as the auto registration of block 630. The Toolboxprovides point cloud registration, geometrical shape fitting to 3-Dpoint clouds, and the ability to read, write, store, display, andcompare point clouds, as well as the ability to combine multiple pointclouds to reconstruct a 3-D scene. The Toolbox can implementregistration algorithms usable in the method of FIG. 6 , which are basedon the Iterative Closest Point (ICP) algorithm, the Normal-DistributionsTransform (NDT) algorithm, the phase correlation algorithm, and theCoherent Point Drift (CPD) algorithm, respectively, as will beunderstood by those of skill in the art. Commonly assigned U.S. patentpublication 20200371240 (“REAL-TIME IMAGE FORMATION FROM GEIGER-MODELADAR”), which is incorporated by reference, also describes applying theICP algorithm in LADAR processing.

Referring again to FIGS. 1, 5A, and 5B, after registration (block 630),the registered point clouds are merged into a single conglomerate, whichin this embodiment corresponds to a single volumetric image made up ofdistinct parts (the registered point clouds) (block 635). The localspatial density in the volumetric image is computed (block 640), andthen a non-liner scaling is applied (block 645) to the local spatialpoint density. As explained further herein, the process steps 640-650may not be necessary in all embodiments, especially if the results arenot displayed to a human observer. The resulting image, with non-linearscaling, can be displayed, if needed, such as output device 112 (FIG. 1); alternately, the resulting image can be provided to another entityprocess for further action. The output of the multi-dwell processing is,thus, a multi-dwell image 408 (FIG. 5A), which results from combiningthe data 402 from disparate, non-contiguous sub-dwells, after theprocessing of blocks 605-650, into a 6D image with improved SNR ascompared to an image based on only one dwell (e.g., a long dwell) (block655).

An advantage of the method of FIG. 6 is its ability to get higherfidelity images in less time. In prior art methods, to obtain a highfidelity image, it can be necessary to devote a large amount ofcontiguous aperture time through that process, which can make it very“expensive” from an operational point of view - meaning, the time spentusing the processor, transmitter, receiver, sensors, etc., to processthe longer continuous aperture time, is taken away from allowing asystem to do other things with those components and modules in thesystem. The method of FIG. 6 enables high fidelity imaging with a largernumber of small dwells on the target, which is much more convenient tothe overall system in terms of its operation and timeline.

Yet another advantage of the method of FIG. 6 is that this method alsocan create/provide 3D images from data that was taken incidentally whiledoing other things, e.g., data that wasn’t necessarily coordinated intime as part of a planned set of dwells intended to image a specificarea or object. For example, the method of FIG. 6 is usable for datathat is separated in time, which data may have been a byproduct oftaking data for other reasons. The method of FIG. 6 can be used, forexample, to locate an unexpected image of something that can bediscerned or detected by looking through various sets of data taken atdifferent times, by different entities, wherein, after analysis, atarget or particular target motion can be found or even discovered, viaanalysis of the data in accordance with the method of FIG. 6 (along withmethods such as that of FIG. 4A, which can create 3D velocity data), tocreate a resulting 3D image of a target and also its motion, includingtargets whose motion is not known and/or was not expected, and/ortargets whose existence or motion may be “buried” in other data,including discontinuous data. The method of FIG. 6 thus can be appliedto image data taken via many other types of existing techniques,including data taken at separate times, especially data from any processthat is configured to output 3D velocity information about a photoevent.

FIG. 7 is a set of graphs 702-712 illustrating an exemplary operation ofthe method of FIG. 6 to produce single-dwell state estimates, inaccordance with one embodiment. FIG. 7 builds on the previous example ofFIG. 2 -4E. Consider a set of five 150 ms noncontiguous dwells,scattered throughout the target’s 1-second period-of-motion. Forexample, in FIG. 7 , in plot 702, the 5150 ms noncontiguous dwells areshown, e.g., as the asterisks 722, 724, 726, 728, 730 that follow alongthe curve 732, are center points at the times the measurements are taken(this is similar for the other coordinates shown in plots 704, 706, 708,710, 712). As FIG. 7 shows, at t = 0 seconds, at the first “dwell” 722,the approximate position of the target is at x=-5 m, y= -20 m, andestimated R=6845 m. By the time of the second “dwell” 724, at t = 0.2 s,the approximate dimensions are x = -20 m, y=-5 m, and estimated R=6875m. As FIG. 7 illustrated, in between each of the dwells, the target ismoving continuously and potentially in a complicated way, where both theposition and the velocity are varying sinusoidally in x and y and wherethe object is accelerating in the range direction. And the object isdoing this continuously, even though the dwells are only being taken atdiscrete intervals. The solid line labeled “truth” in each plot, is theobject’s actual trajectory. In accordance with FIGS. 4A and 6 , the SSCprocess of FIG. 4A (and block 610 of FIG. 6 ) operates on each dwellindependently, generating detections for each. Within the uncertaintybounds, the state-space position of the bi-sphere (FIG. 2 ) is extractedaccurately.

In accordance with some embodiments, herein, the multi-dwell processingof FIG. 6 can be applied to the sets of detections of FIG. 7 , to createa merged point cloud. This merged output has significantly higher imageSNR than that of a single dwell. For example, FIGS. 8A-8B are a firstpair of exemplary merged point cloud images comparing a singled dwellimage 800 (FIG. 8A) with a multi-dwell image 850 (FIG. 8B), inaccordance with one embodiment. Note that this result was achieved on anexemplary target with a smoothly-differentiable motion - i.e., thetarget had non-zero components to its velocity, acceleration, jerk, etc.

FIGS. 9A-9B are a second pair of exemplary point cloud images comparinga single dwell point cloud image 900 with object facet model (FIG. 9A)with a multi-dwell image with object facet model (FIG. 9B), inaccordance with one embodiment. FIG. 9A illustrates a single-dwellpoint-cloud image 900 for the original model of FIG. 2 , and FIG. 9Billustrates a multi-dwell point-cloud 950 for the original model of FIG.2 . As a comparison of FIG. 9B vs FIG. 9A shows, compared against theoriginal model (right figure), the multi-dwell imagery of FIG. 9B hashigher SNR, better shape fidelity, and wider coverage than single-dwellimages. In FIG. 9B, the point cloud 924 represents the output of block650 of FIG. 6 , and the object facet model 922 is the actual shape ofthe target. As can be seen, point cloud 924 of FIG. 9B, that resultsfrom the multi-dwell process of FIG. 6 , is a better approximation ofthe actual shape than is the point cloud 906 of FIG. 9A (single dwell).

Another aspect to note in FIGS. 9A and 9B are the specific dense regionslabeled as 906 in FIG. 9A (single dwell) and 926 in FIG. 9B (multidwell). These areas are at a higher density because there is a greaterdensity of points in those regions (greater local density), because, forthe example object of FIG. 2 , there is a bigger surface of which thetransmitted pulses can reflect. Thus, a greater number of pointscorresponds to a greater density, and this is an area of the image wherea process step like determining local spatial point density (block 640)and applying non-linear scaling to the local spatial point density(block 645) provide advantages. Without scaling for local spatialdensity, it can be more difficult to “see” that one is lookingat/detecting a surface. In particular, for a human observer, it can bedifficult to detect such spatial density, and it may merely look like acluster of points in a region. This scaling can be advantageous if theoutput of the method of FIG. 6 is to be displayed to a human observer,to help present the findings in a way a human observer can see. However,if the output of the method of FIG. 6 is to another process or entitythat does not need to display the image to a human observer, it may notbe necessary to perform the blocks 640-650, as the next entity orprocess may be able to discern spatial point density in other ways.

As can be seen in the above descriptions of various embodiments herein,in certain embodiments, the multi-dwell process herein (e.g., of FIG. 6) leverages the novel approach to acquisition utilized by theState-Space Carving of FIG. 4A and applies this technique to imageformation. In certain embodiments, because both functions are performedsimultaneously, overall system resources are employed more efficiently,increasing capacity and reducing latency. Further, in certainembodiments, the use of disjoint dwells provides maximum flexibility inaperture time scheduling. In addition, in certain embodiments, the useof 6D segmentation enables application of the techniques herein tocomplex non-rigid scenes.

It is envisioned that any or all of the embodiments described hereinand/or illustrated in FIG. 1 -9B herein could be combined with and/oradapted to work with the technologies described in one or more of thecommonly assigned U.S. Pat. Applications and patents that have beenlisted above and additionally , including but not limited to:

-   U.S. Pat. Applications: SN 16,863,064 (“SYSTEM AND METHOD FOR    DETERMINING RANGE-RATE AND RANGE EXTENT OF A TARGET)”; SN 17/138,365    (“VIDEO-TRACKING OF SPARSE GEIGER MODE DATA”); 17/138,386 (“SIX    DIMENSIONAL TRACKING OF SPARSE LADAR DATA”)-   U.S. Pat. Publications: 20210341606 (“SYSTEM AND METHOD FOR    CALCULATING A BINARY CROSS-CORRELATION”); 20200371240 (“REAL-TIME    IMAGE FORMATION FROM GEIGER-MODE LADAR”); and 20210341576 (“HIGH    PULSE REPETITION FREQUENCY LIDAR”)-   U.S. Pat.: 10,620,315 (“LADAR RANGE ESTIMATE WITH RANGE RATE    COMPENSATION”); 10371818 (“MOTION COMPENSATION FOR DYNAMIC    IMAGING”); 9858304 (“COMPUTING CROSS-CORRELATIONS FOR SPARSE DATA”);    and 9,335,414 (“FREQUENCY AGILE LADAR”).

Each of the above patents and patent publications is incorporated byreference.

FIG. 10 is a block diagram of an exemplary computer system usable withat least some of the systems and apparatuses of FIG. 1 -9B, inaccordance with one embodiment. In some embodiments, the computer system1000 of FIG. 10 can be usable as the processor 110 of FIG. 1 . Referenceis made briefly to FIG. 10 , which shows a block diagram of a computersystem 1000 usable with at least some embodiments. The computer system1000 also can be used to implement all or part of any of the methods,equations, and/or calculations described herein.

As shown in FIG. 10 , computer 1000 may include processor/CPU 1002,volatile memory 1004 (e.g., RAM), non-volatile memory 1006 (e.g., one ormore hard disk drives (HDDs), one or more solid state drives (SSDs) suchas a flash drive, one or more hybrid magnetic and solid state drives,and/or one or more virtual storage volumes, such as a cloud storage, ora combination of physical storage volumes and virtual storage volumes),graphical user interface (GUI) 1010 (e.g., a touchscreen, a display, andso forth) and input and/or output (I/O) device 1008 (e.g., a mouse, akeyboard, etc.). Non-volatile memory 1004 stores, e.g., journal data1004 a, metadata 1004 b, and pre-allocated memory regions 1004 c. Thenon-volatile memory, 1006 can include, in some embodiments, an operatingsystem 1014, and computer instructions 1012, and data 1016. In certainembodiments, the computer instructions 1012 are configured to provideseveral subsystems, including a routing subsystem 1012A, a controlsubsystem 1012 b, a data subsystem 1012c, and a write cache 1012 d. Incertain embodiments, the computer instructions 1012 are executed by theprocessor/CPU 1002 out of volatile memory 1004 to implement and/orperform at least a portion of the systems and processes shown in FIG. 1-9B. Program code also may be applied to data entered using an inputdevice or GUI 1010 or received from I/O device 1008.

The systems, architectures, and processes of FIG. 1 -9B are not limitedto use with the hardware and software described and illustrated hereinand may find applicability in any computing or processing environmentand with any type of machine or set of machines that may be capable ofrunning a computer program and/or of implementing a LADAR/LIDAR system.The processes described herein may be implemented in hardware, software,or a combination of the two. The logic for carrying out the methodsdiscussed herein may be embodied as part of the system described in FIG.10 . The processes and systems described herein are not limited to thespecific embodiments described, nor are they specifically limited to thespecific processing order shown. Rather, any of the blocks of theprocesses may be re-ordered, combined, or removed, performed in parallelor in serial, as necessary, to achieve the results set forth herein.

Processor 1002 may be implemented by one or more programmable processorsexecuting one or more computer programs to perform the functions of thesystem. As used herein, the term “processor” describes an electroniccircuit that performs a function, an operation, or a sequence ofoperations. The function, operation, or sequence of operations may behard coded into the electronic circuit or soft coded by way ofinstructions held in a memory device. A “processor” may perform thefunction, operation, or sequence of operations using digital values orusing analog signals. In some embodiments, the “processor” can beembodied in one or more application specific integrated circuits(ASICs). In some embodiments, the “processor” may be embodied in one ormore microprocessors with associated program memory. In someembodiments, the “processor” may be embodied in one or more discreteelectronic circuits. The “processor” may be analog, digital, ormixed-signal. In some embodiments, the “processor” may be one or morephysical processors or one or more “virtual” (e.g., remotely located or“cloud”) processors.

Various functions of circuit elements may also be implemented asprocessing blocks in a software program. Such software may be employedin, for example, one or more digital signal processors,microcontrollers, or general-purpose computers. Described embodimentsmay be implemented in hardware, a combination of hardware and software,software, or software in execution by one or more physical or virtualprocessors.

Some embodiments may be implemented in the form of methods andapparatuses for practicing those methods. Described embodiments may alsobe implemented in the form of program code, for example, stored in astorage medium, loaded into and/or executed by a machine, or transmittedover some transmission medium or carrier, such as over electrical wiringor cabling, through fiber optics, or via electromagnetic radiation. Anon-transitory machine-readable medium may include but is not limited totangible media, such as magnetic recording media including hard drives,floppy diskettes, and magnetic tape media, optical recording mediaincluding compact discs (CDs) and digital versatile discs (DVDs), solidstate memory such as flash memory, hybrid magnetic and solid-statememory, non-volatile memory, volatile memory, and so forth, but does notinclude a transitory signal per se. When embodied in a non-transitorymachine-readable medium and the program code is loaded into and executedby a machine, such as a computer, the machine becomes an apparatus forpracticing the method.

When implemented on one or more processing devices, the program codesegments combine with the processor to provide a unique device thatoperates analogously to specific logic circuits. Such processing devicesmay include, for example, a general-purpose microprocessor, a digitalsignal processor (DSP), a reduced instruction set computer (RISC), acomplex instruction set computer (CISC), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA), aprogrammable logic array (PLA), a microcontroller, an embeddedcontroller, a multi-core processor, and/or others, includingcombinations of one or more of the above. Described embodiments may alsobe implemented in the form of a bitstream or other sequence of signalvalues electrically or optically transmitted through a medium, storedmagnetic-field variations in a magnetic recording medium, etc.,generated using a method and/or an apparatus as recited in the claims.

For example, when the program code is loaded into and executed by amachine, such as the computer of FIG. 10 , the machine becomes anapparatus for practicing one or more of the described embodiments. Whenimplemented on one or more general-purpose processors, the program codecombines with such a processor to provide a unique apparatus thatoperates analogously to specific logic circuits. As such ageneral-purpose digital machine can be transformed into a specialpurpose digital machine. FIG. 10 shows Program Logic 1024 embodied on acomputer-readable medium 1020 as shown, and wherein the Logic is encodedin computer-executable code configured for carrying out the reservationservice process of this invention and thereby forming a Computer ProgramProduct 1022. The logic may be the same logic on memory loaded onprocessor. The program logic may also be embodied in software modules,as modules, or as hardware modules. A processor may be a virtualprocessor or a physical processor. Logic may be distributed acrossseveral processors or virtual processors to execute the logic.

In some embodiments, a storage medium may be a physical or logicaldevice. In some embodiments, a storage medium may consist of physical orlogical devices. In some embodiments, a storage medium may be mappedacross multiple physical and/or logical devices. In some embodiments,storage medium may exist in a virtualized environment. In someembodiments, a processor may be a virtual or physical embodiment. Insome embodiments, a logic may be executed across one or more physical orvirtual processors.

For purposes of illustrating the present embodiments, the disclosedembodiments are described as embodied in a specific configuration andusing special logical arrangements, but one skilled in the art willappreciate that the device is not limited to the specific configurationbut rather only by the claims included with this specification. Inaddition, it is expected that during the life of a patent maturing fromthis application, many relevant technologies will be developed, and thescopes of the corresponding terms are intended to include all such newtechnologies a priori.

The terms “comprises,” “comprising”, “includes”, “including”, “having”and their conjugates at least mean “including but not limited to”. Asused herein, the singular form “a,” “an” and “the” includes pluralreferences unless the context clearly dictates otherwise. Variouselements, which are described in the context of a single embodiment, mayalso be provided separately or in any suitable subcombination. It willbe further understood that various changes in the details, materials,and arrangements of the parts that have been described and illustratedherein may be made by those skilled in the art without departing fromthe scope of the following claims.

Throughout the present disclosure, absent a clear indication to thecontrary from the context, it should be understood individual elementsas described may be singular or plural in number. For example, the terms“circuit” and “circuitry” may include either a single component or aplurality of components, which are either active and/or passive and areconnected or otherwise coupled together to provide the describedfunction. Additionally, the term “signal” may refer to one or morecurrents, one or more voltages, and/or or a data signal. Within thedrawings, like or related elements have like or related alpha, numericor alphanumeric designators (e.g., a component labeled as “204” in FIG.2 may be similar to a component labeled “404” in FIG. 4 , etc.).Further, while the disclosed embodiments have been discussed in thecontext of implementations using discrete components, including somecomponents that include one or more integrated circuit chips), thefunctions of any component or circuit may alternatively be implementedusing one or more appropriately programmed processors, depending uponthe signal frequencies or data rates to be processed and/or thefunctions being accomplished.

Similarly, in addition, in the Figures of this application, in someinstances, a plurality of system elements may be shown as illustrativeof a particular system element, and a single system element or may beshown as illustrative of a plurality of particular system elements. Itshould be understood that showing a plurality of a particular element isnot intended to imply that a system or method implemented in accordancewith the invention must comprise more than one of that element, nor isit intended by illustrating a single element that the invention islimited to embodiments having only a single one of that respectiveelements. In addition, the total number of elements shown for aparticular system element is not intended to be limiting; those skilledin the art can recognize that the number of a particular system elementcan, in some instances, be selected to accommodate the particular userneeds.

In describing and illustrating the embodiments herein, in the text andin the figures, specific terminology (e.g., language, phrases, productbrands names, etc.) may be used for the sake of clarity. These names areprovided by way of example only and are not limiting. The embodimentsdescribed herein are not limited to the specific terminology soselected, and each specific term at least includes all grammatical,literal, scientific, technical, and functional equivalents, as well asanything else that operates in a similar manner to accomplish a similarpurpose. Furthermore, in the illustrations, Figures, and text, specificnames may be given to specific features, elements, circuits, modules,tables, software modules, systems, etc. Such terminology used herein,however, is for the purpose of description and not limitation.

Although the embodiments included herein have been described andpictured in an advantageous form with a certain degree of particularity,it is understood that the present disclosure has been made only by wayof example, and that numerous changes in the details of construction andcombination and arrangement of parts may be made without departing fromthe spirit and scope of the described embodiments. Having described andillustrated at least some the principles of the technology withreference to specific implementations, it will be recognized that thetechnology and embodiments described herein can be implemented in manyother, different, forms, and in many different environments. Thetechnology and embodiments disclosed herein can be used in combinationwith other technologies. In addition, all publications and referencescited herein are expressly incorporated herein by reference in theirentirety.

It should be appreciated that individual elements of differentembodiments described herein may be combined to form other embodimentsnot specifically set forth above. Various elements, which are describedin the context of a single embodiment, may also be provided separatelyor in any suitable sub-combination. It should also be appreciated thatother embodiments not specifically described herein are also within thescope of the following claims.

What is claimed is:
 1. A system, comprising: a receiver configured toreceive a plurality of scattered laser pulses, each respective scatteredlaser pulse in the plurality of scattered laser pulses associated withat least one respective dwell; and a processor configured for: receivinga set of three-dimensional (3D) velocity information derived from photoevents detected in information associated with the received plurality ofscattered laser pulses, wherein each respective dwell in the pluralityof dwells is associated with one or more respective photo events, andwherein the 3D velocity information comprises information estimatingeach respective photo event’s respective position in six-dimensional(6D) space during the respective dwell associated with the respectivephoto event; projecting, for each dwell, respective photo events of thedwell into a common reference frame, wherein the common reference frameis determined based on the 3D velocity information, to generate a set ofmotion-compensated point clouds, the set of motion-compensated pointclouds comprising at least one motion-compensated point cloud for atleast one respective dwell; registering each respectivemotion-compensated point cloud, for each dwell, to the othermotion-compensated point clouds in the set of motion-compensated pointclouds, to generate a set of registered point clouds; and merging theset of registered point clouds into a volumetric image.
 2. The system ofclaim 1, wherein the processor is further configured for: computing, forthe volumetric image, a local spatial point density; and applying anon-linear scaling to the local spatial point density to form a scaledvolumetric image.
 3. The system of claim 2, wherein the processor isfurther configured for displaying the scaled volumetric image.
 4. Thesystem of claim 1, wherein the plurality of dwells is noncontiguous. 5.The system of claim 1, wherein the plurality of dwells is separated intime.
 6. The system of claim 1, wherein the set of 3D velocityinformation is generated using a state space carving (SSC) process. 7.The system of claim 1, wherein the plurality of scattered pulses isassociated with a single photon laser detection and ranging (LADAR)(SPL) system.
 8. The system of claim 1, wherein the scattered laserpulses and detected photo events are associated with a target andwherein the common reference frame comprises an instantaneous referenceframe that is associated with the target and which is based on the setof 3D velocity information.
 9. The system of claim 8, wherein the targetassociated with the plurality of scattered laser pulses, has smoothlydifferentiable motion.
 10. The system of claim 1 wherein the processoris configured for registering each respective motion-compensated pointcloud to correct at least one of translation and orientation errors, inat least a portion of the set of motion-compensated point clouds. 11.The system of claim 1, wherein the processor is configured forregistering each respective motion-compensated point cloud using aniterative closest point (ICP) algorithm.
 12. A method, comprising:receiving a plurality of scattered laser pulses, each respectivescattered laser pulse in the plurality of scattered laser pulsesassociated with at least one plurality of respective dwells; receiving aset of three-dimensional (3D) velocity information derived from photoevents detected in information associated with the received plurality ofscattered laser pulses, wherein each respective dwell in the pluralityof dwells is associated with one or more respective photo events, andwherein the 3D velocity information comprises information estimatingeach respective photo event’s respective position in six-dimensional(6D) space during the respective dwell associated with the respectivephoto event; projecting, for each dwell, respective photo events of thedwell into a common reference frame, wherein the common reference frameis determined based on the 3D velocity information, to generate a set ofmotion-compensated point clouds, the set of motion-compensated pointclouds comprising at least one motion-compensated point cloud for atleast one respective dwell; registering each respectivemotion-compensated point cloud, for each dwell, to the othermotion-compensated point clouds in the set of motion-compensated pointclouds, to generate a set of registered point clouds; and merging theset of registered point clouds into a volumetric image.
 13. The methodof claim 12, further comprising displaying the volumetric image as ascaled volumetric image.
 14. The method of claim 12, further comprisinggenerating the set of 3D velocity information using a state spacecarving (SSC) process.
 15. The method of claim 12, wherein the scatteredlaser pulses and detected photo events are associated with a target andwherein the common reference frame comprises an instantaneous referenceframe that is associated with the target and which is based on the setof 3D velocity information.
 16. The method of claim 12, wherein theregistering each respective motion-compensated point cloud is configuredto correct at least one of translation and orientation errors, in atleast a portion of the set of motion-compensated point clouds.
 17. Ameans for laser detection, comprising: means for receiving a pluralityof scattered laser pulses, each respective scattered laser pulse in theplurality of scattered laser pulses associated with at least oneplurality of respective dwells; means for receiving a set ofthree-dimensional (3D) velocity information derived from photo eventsdetected in information associated with the received plurality ofscattered laser pulses, wherein each respective dwell in the pluralityof dwells is associated with one or more respective photo events, andwherein the 3D velocity information comprises information estimatingeach respective photo event’s respective position in six-dimensional(6D) space during the respective dwell associated with the respectivephoto event; means for projecting, for each dwell, respective photoevents for the dwell into a common reference frame, wherein the commonreference frame is determined based on the 3D velocity information, togenerate a set of motion-compensated point clouds, the set ofmotion-compensated point clouds comprising at least onemotion-compensated point cloud for at least one respective dwell; meansfor registering each respective motion-compensated point cloud, for eachdwell, to the other motion-compensated point clouds in the set ofmotion-compensated point clouds, to generate a set of registered pointclouds; and means for merging the set of registered point clouds into avolumetric image.
 18. The means for laser detection of claim 17 furthercomprising means for generating the set of 3D velocity information usinga state space carving (SSC) process.
 19. The means for laser detectionof claim 17, wherein the scattered laser pulses and detected photoevents are associated with a target and wherein the common referenceframe comprises an instantaneous reference frame that is associated withthe target and which is based on the set of 3D velocity information. 20.The means for laser detection of claim 17, wherein the registering eachrespective motion-compensated point cloud is configured to correct atleast one of translation and orientation errors, in at least a portionof the set of motion-compensated point clouds.